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Abstract 


Typically,  the  development  of  tools  and  systems  for  the  military  is  requirement  driven; 
systems  are  developed  to  meet  specified  requirements  and  evaluated  on  compliance 
with  those  requirements.  The  real  question  we  should  ask  about  tools  and  systems  in 
development  is,  “what  benefit  does  the  system  provide  to  the  warfighter?”  The  U.S. 
Army  Topographic  Engineering  Center  (TEC)  is  sponsoring  a  series  of  rigorous 
experiments  designed  to  answer  this  question  and  thereby  help  to  focus  its  research  and 
development  efforts.  The  first  experiment  in  this  series,  which  was  presented  at  the 
12th  ICCRTS,  demonstrated  the  value  of  a  Geospatial  Decision  Support  Systems 
(GDSS),  Battlespace  Terrain  Reasoning  and  Awareness  -  Battle  Command  (BTRA- 
BC),  in  a  strictly  terrain  analysis  scenario.  This  second  experiment  in  the  series, 
building  upon  the  results  of  the  first  experiment,  evaluated  the  value  of  BTRA-BC  in  a 
realistic  planning  environment  with  a  scenario  that  requires  more  complex  decision¬ 
making.  This  paper  discusses  the  experimental  design  (presented  at  the  13th  ICCRTS) 
and  the  preliminary  results  of  this  experiment. 


1.  Background 


As  researchers  and  developers  strive  to  provide  advanced  tools  to  process  faster 
and  more  accurate  data,  it  necessitates  the  assessment  of  each  innovation  so  that  key 
resources  can  be  allocated  to  areas  that  yield  the  most  “bang  for  the  buck.”  The  Joint 
Geospatial  Enterprise  Services  (JGES)  program  of  the  U.S.  Army  Engineer  Research  and 
Development  Center  (ERDC)  is  designed  to  meet  this  need  by  evaluating  the  value-added 
to  military  decision-making  through  the  use  of  Geospatial  Decision  Support  Systems 
(GDSSs).  We  define  a  GDSS  as  a  Decision  Support  System  that  performs  automated 
analyses  of  geospatial  data,  and  then  displays  that  geospatial  data  and  the  results  of 
automated  analyses,  i.e.,  geospatial  information.  A  GDSS  can  consist  of  one  or  more 
Geospatial  Decision  Support  Products  (GDSPs)  which  calculate  and  display  geospatial 
data  or  geospatial  information.  BTRA-BC  consists  of  a  collection  of  GDSPs  called 
Tactical  Spatial  Objects  (TSOs)  that  can  be  embedded  as  modules  of  a  GDSS.  TSOs  are 
computationally  lightweight  objects  that  provide  geospatial  information  services  specific 
to  the  context  for  which  they  are  developed.  The  context  for  which  our  TSOs  are 
developed  is  that  of  military  ground  operations.  In  that  context,  the  potentially  superior 
situation  awareness  afforded  by  BTRA-BC  TSOs  opens  up  new  possibilities  for  the 
conduct  of  military  operations.  Translating  these  possibilities  into  practical  decision 
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support  requires  a  build-test-build  cycle  that  channels  technology  in  spiral  development 
to  ensure  results  that  best  support  the  warfighter.  This  paper  reports  on  the  second  in  a 
series  of  experiments  designed  to  assess  the  value  of  Geospatial  information  to  the 
decision-maker,  and  to  provide  results  that  inform  the  spiral  development  cycle.  In  the 
case  of  military  GDSSs,  the  ultimate  decision  maker  is  the  military  commander,  and  the 
ultimate  goal  is  to  support  command  decisions  in  the  most  effective  way. 

The  first  experiment  in  the  series  assessed  the  value  of  the  Battlespace  Terrain  Reasoning 
and  Awareness  -  Battle  Command  (BTRA-BC)  GDSS  to  the  terrain  analyst  performing 
the  Intelligence  Preparation  of  the  Battlefield  (IPB)  planning  task  using  the  Anny’s 
Digital  Topographic  Support  System  (DTSS).  Army  and  Marine  Corps  enlisted  terrain 
analysts  attending  the  Advanced  Terrain  Analyst  Course  (AT AC)  were  given  planning 
tasks  to  complete  with  and  without  BTRA-BC  functionality.  The  experimental  design 
was  very  similar  to  that  of  our  current  experiment  (described  in  section  4  below).  The 
results  of  the  experiment  demonstrated  that  using  BTRA-BC  functionality  (1) 
dramatically  decreased  the  time  to  perform  the  terrain  analysis  tasks,  (2)  improved  the 
quality  of  the  terrain  analysis  output,  (3)  maintained  or  improved  the  participants’ 
knowledge  of  the  impact  of  terrain  on  planning,  (4)  maintained  or  reduced  the  variability 
in  both  time  to  completion  and  plan  quality,  and  (5)  improved  the  participants’  perception 
of  the  quality  of  their  plans. 

The  goal  of  the  current  experiment  is  to  continue  to  assess  the  benefits  of  BTRA- 
BC  information  and  knowledge  products.  Based  on  what  was  learned  from  the  first 
experiment,  this  experiment  explored  the  benefits  of  GDSSs  for  planners  who  perform 
more  complex  military  decision-making  tasks.  Because  BTRA-BC  provides  geospatial 
decision  support,  the  automated  analysis  functionality  is  of  primary  interest.  Our  current 
experiment  was  specifically  designed  to  assess  these  capabilities.  The  BTRA-BC 
capabilities  evaluated  in  this  study  include  all  the  factors  involved  in  the  previous 
experiment  as  well  as  the  ability  of  BTRA-BC  to  identify  battle  positions,  engagement 
areas,  and  assist  the  planner  in  determining  optimum  ambush  sites  and  Named  Areas  of 
interest. 
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The  paper  is  organized  as  follows.  Section  2  describes  the  overall  scope  of  our 
research  program  and  the  scope  of  this  experiment.  Section  3  discusses  the  primary  and 
secondary  hypotheses  to  be  examined.  Section  4  lays  out  the  design  of  the  experiment 
and  the  reasoning  that  led  to  this  design.  Section  5  discusses  the  computing  environment 
to  be  used  in  the  experiment.  Section  6  describes  the  metrics  to  be  used  to  quantify  the 
results  of  each  trial.  So  far,  the  experiment  has  been  conducted  with  only  half  the 
planned  number  of  participants  (8)  that  we  anticipate  will  be  needed  to  generate 
statistically  significant  results  for  most  of  the  hypotheses.  Section  7  and  8  discusses  the 
preliminary  results  and  our  tentative  conclusions. 

2.  Scope  of  Experiment 

Our  ultimate  objective  was  to  evaluate  the  benefit,  to  commanders  at  the  brigade 
level  and  below,  of  combining  a  fully  developed  GDSS  with  currently  available 
Command  and  Control  planning  tools.  The  first  experiment,  presented  at  12th  ICCRTS 
(Laskey,  et.al.,  2007),  was  limited  in  scope  to  the  terrain  analysis  portion  of  Intelligence 
Preparation  of  the  Battlefield  (IPB)  process.  Our  current  experiment  builds  on  the  first 
experiment  by  expanding  the  scope  to  include  the  more  complex  decision-making 
required  to  develop  a  Course  of  Action  (CO  A).  In  this  experiment,  the  general  scenario 
asked  military  planners,  working  individually,  to  plan  a  battalion  movement  to  seize  an 
objective  in  the  presence  of  enemy  units.  Follow-on  experiments  will  further  address 
different  kinds  of  planning  problems,  at  various  levels  of  command,  and  involving 
collaboration  among  members  of  staffs  as  well  as  individual  decision  makers. 

3.  Hypotheses 

As  we  discovered  while  planning  the  first  experiment,  in  order  to  evaluate  the 
military  value  of  GDSPs  in  general  and  specifically  BTRA-BC,  we  needed  a  clear 
definition  of  military  value,  along  with  quantifiable  metrics  of  value.  Our  determination 
of  what  constitutes  value  in  this  experiment  is  based  on  discussions  with  several 
experienced  military  planners.  These  planners  believe  that  the  value  of  GDSSs  lie  in 
their  ability  to: 
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(1)  Reduce  the  time  spent  generating  a  given  tactical  decision  product.  Since  the 
timeframe  available  to  military  decision  makers  is  limited,  GDSSs  that  reduce  the 
time  required  to  produce  the  desired  output  can  free  up  time  for  a  more  thorough 
analysis  of  the  large  amount  of  data  available.  This  more  complete  analysis  is 
expected  to  result  in  a  higher  quality  output  that  will  be  of  more  value  to  the 
decision  maker. 

(2)  Automate  many  of  the  routine  planning  tasks.  Many  of  the  terrain  evaluation 
tasks  traditionally  performed  by  military  planners  with  paper  maps  and  acetate 
overlays  are  sufficiently  rote  in  nature  that  a  GDSS,  given  digital  information  and 
the  appropriate  parameters,  can  perform  these  low-level  terrain  analysis  functions 
more  quickly  and  with  less  error  than  a  human. 

(3)  Provide  standardized  outputs.  A  well-designed  GDSS  should  provide  a  more 
flexible,  precise,  and  easily  understood  display  of  the  infonnation  required.  It 
should  display  data  and  the  results  of  analyses  in  a  more  readily  understandable 
format  than  idiosyncratic  manual  notation. 

A  danger  of  automation  is  that  exclusive  reliance  on  a  tool  for  analysis  of  data 
might  reduce  analyst  familiarity  with  the  terrain  and  its  impact  on  military  planning.  In 
response  to  this  concern,  our  experts  believe  that  the  automated  analyses  conducted  by 
the  GDSS  are  procedural  and  that  using  the  output  of  the  GDSS  will  not  compromise  the 
level  of  understanding  by  the  analyst.  The  experiment  is  designed  to  test  this  belief. 

It  follows  from  the  discussion  above  that,  in  comparison  with  decision-makers 
using  tools  without  BTRA-BC  functionality,  we  hypothesize  that  trained  and  experienced 
military  planners  who  use  BTRA-BC  would: 

1.  Produce  a  CO  A  more  quickly.  Rationale:  The  automation  and  analysis 
functions  in  BTRA-BC  should  allow  the  participants  to  complete  the 
repetitive  and  rote  tasks  more  quickly  allowing  more  time  for  the 
generation  of  more  options  for  the  COA  and  a  subsequently  higher  quality 
product. 
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2. 


Produce  a  higher  quality  output.  Rationale:  The  automaton  in  BTRA-BC 
should  minimize  errors  of  omission  and  calculation.  Furthermore,  the 
standardized  graphical  representation  of  important  terrain  features  and 
TSOs  will  display  information  more  succinctly. 

3.  Display  as  good  an  understanding  of  the  impact  of  the  given  terrain  on 
military  decision-making.  Rationale:  The  judgment  required  to  complete 
the  required  tasks  will  still  be  required  when  using  BTRA-BC. 

As  the  determination  of  military  value  and  the  design  of  the  experiment  evolved, 
we  identified  several  secondary  hypotheses.  The  automation  of  previously  manual  tasks, 
which  adds  value  to  using  a  GDSS,  would  likely  reduce  the  variation  in  the  output.  As 
this  reduction  in  variation  does  not  necessarily  add  value,  this  was  not  considered  a 
primary  hypothesis.  The  structure  of  the  experiments  requires  the  repetition  of  various 
tasks  and  there  was  concern  that  a  learning  effect  my  might  skew  the  results  of  the 
experiment.  The  secondary  hypotheses  investigated  include: 

4.  The  output  generated  with  BTRA-BC  would  be  more  uniform  i.e.  have  less 
variance  in  the  first  two  of  the  three  categories  above  (speed  and  quality), 
than  output  generated  without  the  use  of  BTRA-BC.  Rationale:  Less 
variation  in  the  output  when  using  BTRA-BC  is  expected  due  to  the  level 
of  automation  incorporated  in  BTRA-BC. 

5.  There  would  not  be  a  learning  effect  due  to  experimental  design. 
Rationale:  The  participants  have  previous  training  and  extensive 
experience  using  C2  planning  tools  and  the  tasks  the  participants  are  asked 
to  perform  are  those  that  they  have  performed  in  the  normal  course  of  their 
duties. 

6.  Participants  would  consider  using  BTRA-BC  superior  with  respect  to 
speed,  quality,  usability,  terrain  understanding,  and  overall.  Rationale: 
The  participants  should  consider  BTRA-BC’s  automated  tools  of  benefit 
in  the  planning  process. 
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4.  Study  Design 

The  study  design  employed  a  factorial  design  with  three  independent  variables: 
System  (with  and  without  BTRA-BC  functionality),  System  Order  (whether  the  first 
scenario  was  worked  with  or  without  BTRA-BC  functionality),  and  Scenario  Order 
(whether  scenario  1  or  2  was  worked  first).  System  was  a  within-subjects  variable 
because  each  participant  worked  one  planning  scenario  with  BTRA-BC  and  one  scenario 
without  BTRA-BC.  A  within-subjects  design  was  particularly  valuable  when  the  number 
of  available  participants  is  limited,  as  in  the  current  case.  Results  from  the  sets  of  tasks 
were  compared  for  each  participant,  thus  eliminating  participant-specific  effects  that 
might  add  variability  to  the  results.  System  Order  and  Scenario  Order  were  between- 
subjects  variables  because  any  given  participant  can  only  be  in  one  ordered  sequence  for 
these  variables. 

The  participants  performed  the  same  tasks  on  two  similar  military  planning 
scenarios,  using  the  same  underlying  planning  system,  the  Commander’s  Support 
Environment  (CSE).  One  of  the  tasks  was  performed  with  BTRA-BC  functions  added  to 
the  basic  CSE  functions;  and  the  other  task  was  performed  with  the  basic  CSE  alone. 
The  two  trials  were  essentially  identical  except  for  the  use  of  BTRA-BC  functions.  The 
order  of  the  tasks  was  randomly  selected  so  that  half  of  the  participants  performed  each 
of  the  tasks  first.  Randomizing  the  order  of  the  tasks  enabled  the  analysis  to  control  for 
learning  effects. 

The  instructions,  tasks,  requested  outputs,  and  evaluation  of  these  outputs  were 
the  same  in  both  scenarios  with  the  exception  of  geographic  references  necessitated  by 
the  requirement  to  have  different  geographic  areas  for  each  trial.  Different  geographic 
areas  prevented  participants  from  repeating  their  responses  from  the  first  scenario  when 
they  formed  responses  for  the  second  scenario.  The  two  areas  were  carefully  selected  for 
their  geographic  similarity  such  that  the  tasks  perfonned  by  the  participants  and  the 
expected  results  will  be  as  nearly  identical  as  possible.  Randomization  was  used  to 
control  for  differences  between  scenarios. 

The  participants  were  Army  officers  who  have  been  previously  trained  in  military 
planning  and  have  operational  experience  at  the  battalion  (Bn)  or  brigade  (Bde)  level. 
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Four  of  the  eight  were  active  duty  majors  and  four  were  retired  lieutenant  colonels  and 
colonels.  The  more  junior  of  the  participants  were  very  conversant  with  computer- 
assisted  decision-making  whereas  the  two  of  the  more  senior  were  not  so  conversant. 
Based  on  the  results  of  questionnaires  detailing  the  participants’  training  and  experience, 
the  participants  were  split  into  two  groups  that  are  evenly  balanced  with  respect  to  ability 
and  knowledge.  Group  I  performed  the  set  of  tasks  first  without  BTRA-BC  and  then  with 
BTRA-BC.  Group  II  performed  the  tasks  in  reverse  order.  The  groups  were  further 
divided  into  two  subgroups  while  maintaining  the  balance  of  ability  and  knowledge. 
Each  subgroup  performed  the  same  tasks  for  the  same  two  scenarios,  but  the  two 
subgroups  saw  the  two  scenarios  in  the  opposite  order.  This  design  allowed  us  to  control 
for  differences  due  to  the  order  of  system  use  and  the  scenario  order. 

The  tasks  consisted  of  that  portion  of  the  military  planning  process  beginning  with 
analyzing  the  specific  terrain  given  a  Combined  Obstacle  Overlay  (COO)  up  to  the  point 
of  generating  a  potential  COA.  Specific  tasks  included  (1)  identifying  Mobility  Corridors 
(MC),  (2)  categorizing  MCs  by  size,  (3)  grouping  MCs  to  form  potential  Avenues  of 
Approach  (AA),  (4)  evaluating  enemy  COAs,  (5)  planning  routes  for  3  vehicle  types,  (6) 
identifying  choke  points  on  potential  AAs,  (7)  calculating  transit  times,  (8) 
recommending  subordinate  Areas  of  responsibility,  in  this  case  recommending  company 
battalion  boundaries,  (9)  planning  company  movement  from  a  line  of  departure  (LD)  to 
an  objective  (OBJ),  (10)  recommending  Named  Areas  of  Interest  (NAI),  (1 1)  identifying 
ambush  sites,  (12)  identifying  battle  positions,  and  (13)  generating  a  formatted 
Operations  Order  (OPORD). 

The  participants  produced  a  graphic  overlay  depicting  the  results  of  the  above 
tasks  as  well  as  a  written  OPORD  that  explains  the  reasoning  behind  their  decisions.  In 
order  to  gather  data  for  the  metrics  below,  the  participants  also  responded  to 
questionnaires:  one  that  assessed  their  understanding  of  the  effects  of  terrain  on  the 
military  planning  process  and  one  that  assessed  their  subjective  experience  with  and 
without  BTRA-BC  functionality. 

Prior  to  beginning  the  tasks,  both  groups  of  participants  received  standardized 
training  on  the  use  of  BTRA-BC  and  CSE.  The  training  was  sufficient  to  allow  the 
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participants  to  perform  the  required  tasks  and  included  training  on  the  modes  and  features 
unique  to  BTRA-BC  and  CSE.  The  last  phase  of  the  training  required  the  participants  to 
perform  tasks  based  on  the  training  and  similar  to  those  that  the  participants  encountered 
during  the  trials,  but  of  lesser  complexity. 

5.  Environment 

The  evaluation  was  conducted  using  the  Commanders  Support  Environment 
(CSE)  as  the  Command  and  Control  (C2)  planning  system.  CSE  is  a  robust  C2  planning 
and  execution  based  system  developed  for  experimentation.  CSE  has  been  enhanced  to 
incorporate  the  BTRA-BC  GDSPs.  CSE  provided  the  capability  to  develop  one  or  more 
COAs  through  a  graphically  oriented  interface  to  represent  the  units,  control  measures, 
and  their  tasks.  CSE  was  originally  developed  for  Defense  Advanced  Research  Projects 
Agency  (D  ARP  A)/ Army  Multi-Cell  and  Dismount  C2  Program  (M&D  C2)  which  was 
continued  from  the  Future  Combat  System  Command  &  Control  (FCS  C2)  program. 

CSE  is  primarily  written  in  C++  code  for  the  Microsoft  Windows  environment.  It  is  built 
upon  the  Viecore  FSD  Decision  Support  System  (VDSS),  and  the  Data  Analysis  and 
Visualization  Infrastructure  for  C4i  (Davinci)  Toolkit.  The  VDSS  architecture  enables 
rapid  addition  of  modules  for  communication  between  CSE  and  other  systems  and 
components.  The  CSE’s  GIS  components  are  built  upon  the  Commercial  Joint  Mapping 
Toolkit(C/JMTK)  which  includes  ESRI’s  ArcGIS  Desktop  licensed  at  the  ArcEditor 
level. 

The  CSE  provided  two  main  GDSPs:  BTRA-BC  Movement  Projection  engine 
and  optimized  Line  of  Sight  (LOS)  analysis  in  addition  to  displaying  BTRA-BC  TSOs. 
The  BTRA-BC  Movement  Projection  engine  provided  movement  and  route  analysis. 

This  tool  allowed  the  planners  to  generate  various  types  of  routes  for  his  maneuver 
planning.  These  routes  were  incorporated  into  tasks  that  became  part  of  the  plan.  The 
LOS  GDSP  displays  a  real-time  analysis  based  on  the  relevant  digital  elevation  data. 

CSE  can  also  invoke  and  display  various  BTRA-BC  TSOs.  To  keep  things  simple  for 
this  experiment,  we  utilized  some  pre-generated  TSOs. 
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NORM 


Choke  Point  TSO 


PL  SAM 


OBJ  BLUE  \  \ 

0bstaales\S0 


OBJ  ORANGE 


PL  DAVE 


Water  Crossing  TSO 


Figure  1  is  an  example  of  a  geospatial  product  consisting  of  several  layers 
including  BTRA-BC  TSOs.  The  background  is  an  image  of  a  map  geo-referenced  to  the 
digital  data  on  which  the  graphics  are  based.  The  second  layer  consists  of  operational 
boundary  and  unit  graphics.  Additional  layers  consist  of  TSOs.  The  TSOs  generate  the 
information  from  the  underlying  digital  geospatial  data.  In  Figure  1,  a  route  generated  by 
the  Route  TSO  is  indicated  by  the  blue  line.  Water  crossings  are  outlined  in  green;  choke 
points  for  Bn-sized  units  are  outlined  in  red;  and  natural  obstacles  are  in  solid  red. 

Figures  2  and  3  below  depict  the  same  geographical  region  with  BTRA-BC  TSOs  and 
without  BTRA-BC  TSOs,  respectively. 
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Figure  2:  With  BTRA-BC  TSOs 


6.  Metrics 

Due  to  the  differences  in  the  graphical  representation  between  CSE  alone  and 
CSE  with  BTRA-BC,  blind  scoring  was  not  feasible.  BTRA-BC’s  graphics  were  not 
easily  replicated  manually  in  CSE  and,  although  the  evaluators  were  not  told  which  of  the 
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products  were  produced  using  BTRA-BC,  given  the  number  of  products  they  graded, 
they  were  able  to  determine  which  were  produced  with  BTRA-BC.  Although  the  outputs 
are  distinguishable  as  to  their  source,  the  evaluators  were  Subject  Matter  Experts  (SMEs) 
who  had  no  connection  to  the  development  of  BTRA-BC,  and  we  assumed  the  subjective 
evaluations  are  unbiased. 

The  criteria  for  evaluation  of  the  BTRA-BC  tools  were:  (1)  a  comparison  of  the 
rapidity  with  which  the  requested  outputs  can  be  produced,  (2)  the  quality  of  those 
outputs,  (3)  the  level  of  understanding  of  the  participants  of  the  impact  of  the  terrain  on 
the  military  decision  making,  and  (4)  the  perception  of  the  participants  regarding  the 
merits  of  the  additional  BTRA-BC  functionality. 

Time  to  Completion.  The  evaluation  of  how  quickly  the  desired  outputs  are 
produced  was  measured  objectively  and  independently  of  the  experimental  condition  by 
logging  the  amount  of  time  it  takes  participants  to  complete  the  tasks.  The  maximum 
duration  of  each  trial  was  3.5  hours.  The  actual  time  was  calculated  by  taking  the 
difference  between  the  start  and  stop  times  and  subtracting  any  break  time. 

Quality.  We  considered  two  factors  that  contribute  to  the  quality  of  a 
participant’s  output:  (1)  the  information  presented  and  (2)  how  the  information  was 
presented.  The  method  of  scoring  quality  was  subjective  scoring;  scores  for  each 
participant’s  output  were  assigned  by  independent  Subject  Matter  Experts  (SMEs).  The 
SMEs  judged  the  quality  of  the  output  with  respect  to  the  usefulness  to  the  commander 
and  based  their  evaluations  on  criteria  developed  beforehand.  The  focus  of  the  subjective 
evaluation  was  relatively  broad  but  focused  on  the  BTRA-BC  TSOs  being  evaluated.  The 
participants’  outputs  were  scored  in  fifteen  categories  with  a  total  of  29  sub-categories 
that  encompassed  both  the  graphic  and  written  portions  of  each  output.  The  scoring  was 
done  on  a  5-point  Likert  scale. 

Terrain  Knowledge.  We  administered  a  questionnaire  to  evaluate  the 
participants’  knowledge  of  the  terrain  and  understanding  of  the  impact  of  the  specific 
terrain  on  military  decision-making.  The  answers  to  the  questions  were  not  outputs  of 
CSE  or  BTRA-BC.  Answering  the  questionnaire  required  judgment  and  reasoning  about 
the  terrain  and  its  effect  on  the  military  decision  making,  and  not  just  regurgitating  data 
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presented  by  the  system  (CSE  with  or  without  BTRA-BC).  Like  the  subjective 
evaluation,  the  SMEs  evaluated  the  participants’  answers  on  a  5-point  Likert  scale.  The 
questions  addressed  reasoning  about  the  general  geography  of  the  area,  vehicle  routing 
considerations,  the  selection  of  battle  positions,  engagement  areas,  and  ambush  sites. 


7.  Results. 

We  are  presenting  the  preliminary  results  for  the  first  run  of  the  experiment, 
which  was  conducted  in  April  of  2008  with  eight  participants.  Given  eight  similar 
participants,  the  results  discussed  below  that  are  statistically  significant  with  only  eight 
participants  are  likely  to  remain  significant,  and  results  that  are  not  statistically 
significant  may  become  significant  with  an  additional  eight  participants.  The 
qualifications  set  for  future  participants 
should  result  in  participants  with  similar 
characteristics  to  the  first  eight.  We 
conducted  simulations  (discussed  below)  to 
estimate  the  quality  of  our  conclusions 


OVERALL 


System 

X 

s 

CSE  w/o  BTRA-BC 

143.0 

17.509 

CSE  w /  BTRA-BC 

147.5 

19.936 

Table  1:  Time  to  Completion  Data  (min) 


Hypothesis  1:  Time  to  Completion.  There  was  no  evidence  that  participants 
completed  the  tasks  more  quickly  when  using  CSE  w/  BTRA-BC  than  when  using  CSE 


w/o  BTRA-BC.  A  repeated  measures 
ANOVA  provided  no  evidence  (p  =  0.573) 
that  there  was,  on  average,  any  difference 
between  the  Time  to  Completion  for  the 

p 

two  systems  (Table  1).  1 

O' 

Hypothesis  2:  Quality.  The  | 

jg 

analysis  of  the  Quality  data  provided 
informative  results.  A  repeated  measures 
ANOVA  analyzing  the  participants’  mean 
scores  for  all  thirteen  criteria  of  Quality 
suggests  that  participants  using  CSE  w/ 
BTRA-BC  produced  higher  quality  outputs 


System  and  System  Order  Interactions 


System  use  order 

Error  Bars:  95%  Cl 


Ligure  4:  Plan  Quality  and  System  Order 
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than  when  using  CSE  alone,  but  the  p-value  of  0.08  does  not  reach  the  traditional  level 
for  statistical  significance.  All  thirteen  criteria  of  Quality  were  developed  in  conjunction 
with  SMEs,  but  of  those  thirteen  criteria,  seven  are  criteria  of  general  plan  quality  and  six 
are  specifically  related  to  the  TSOs  being  evaluated.  A  second  repeated  measures 
ANOVA  on  the  latter  six  criteria  indicated  that  with  respect  to  these  TSOs  there  was 
strong  statistical  evidence  (P  =  0.012)  that  the  participants’  plan  quality  was  superior 
using  CSE  w/  BTRA-BC.  As  the  goal  of  this  experiment  was  to  determine  the  benefits  of 
geospatial  technology,  specifically  the  TSOs,  this  result  is  encouraging. 


Hypothesis  3:  Terrain  Understanding.  As  expected,  the  analysis  of  Terrain 
Understanding  data  is  consistent  with  Hypothesis  3,  that  Terrain  Understanding  when 
using  CSE  w/  BTRA-BC  will  not  be  less  than  when  using  CSE  alone.  A  repeated 
measures  ANOVA  result  shows  no  evidence  (p  =  0.271)  that,  on  average,  the  Terrain 
Understanding  differs;  we  cannot  reject  the  hypothesis  that  there  will  be  no  difference. 


Hypothesis  4:  Uniformity.  Equal 
variance  tests  indicated  that  overall  there 
was  no  evidence  (p  =  0.357),  on  average,  of 
a  difference  in  the  variance  of  Time  to 
Completion.  However,  there  was  some 
statistical  evidence  (p  =  0.096)  that,  on 
average,  the  variance  of  time  to  completion 
was  less  with  the  first  systems  used 
(Table  2). 

Hypothesis  5:  Learning  Effect. 

The  analyses  of  Time  to  Completion  and 
Quality  metrics  showed  evidence  of  a 
possible  learning  effect.  First,  for  Time 
to  Completion  there  was  significant 
evidence  (p  =  0.05)  that  the  participants, 
on  average,  completed  the  tasks  more 
quickly  on  the  second  system  used. 


CSE  w/  BTRA-BC 

X 

s 

Used  First 

163.0 

8.832 

Used  Second 

132.0 

14.445 

CSE  w/o  BTRA-BC 

X 

s 

Used  First 

147.5 

7.594 

Used  Second 

138.5 

24.570 

Table  2:  Time  to  Completion  Variance 


Time  x  System  Order 


200.00 


C  100.00- 


50.00- 


0.00J - 1 - 1 - 

BTRA-CSE  CSE-BTRA 

SystemOrder 

Error  Bars:  95%  Cl 

Figure  5:  Time  and  System  Order 


Figure  5  graphically  show  this  learning  effect.  Second,  for  CSE  w/  BTRA-BC,  there  was 
no  evidence  of  a  learning  effect,  the  evidence  suggests  (p  =  0.09)  that  when  using  CSE 
w/o  BTRA-BC  second,  participants’  plan  quality  was  improved  (Figure  4).  This  effect 
may  have  been  due  to  the  participants  having  seen  the  information  provided  by  the 
BTRA-BC  TSOs  on  their  first  trial,  and  were  trying  to  replicate  manually  the  information 
they  provided  in  their  first  trial. 

Hypothesis  6:  Perception.  Paired  T-Tests  of  participant  responses  demonstrated 
that,  on  average,  there  was  strong  evidence  (p  <  0.001  and  p  =  0.013  respectively)  that 
participants  believed  CSE  w/  BTRA-BC  aided  them  in  producing  better  quality  outputs 
and  that  they  thought  CSE  w/  BTRA-BC  was  superior  overall  (Figure  6).  Additionally, 
there  was  evidence  (p  =  0.06)  that  participants  thought  CSE  w/  BTRA-BC  made 
completing  the  tasks  easier 


Participants'  Subjective  Evaluation  of  BTRA-BC 
[95%  Confidence  Intervals] 
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Figure  6:  Subjective  Perception 
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As  shown  in  Table  3,  paired  T-Tests 
of  the  participants  beliefs  indicated,  on 
average,  that  there  was  evidence  (p  <  0.057) 
that  CSE  w/  BTRA-BC  was  more  useful  for 
four  of  the  eight  tasks  and  overall  (green  at 
right),  and  there  was  weak  evidence  (p  < 

0.111)  that  for  three  of  the  eight  tasks  that 
CSE  w/  BTRA  was  more  useful  (blue  at 
right). 

Predicting  significance  levels  with  16  subjects.  In  order  to  determine  whether 
the  distributions  of  the  responses  from  the  eight  participants  could  be  treated  as  normal, 
an  analysis  of  residuals  was  conducted  on  the  Time  to  Solution,  Quality,  and  Terrain 
Understanding  data.  The  residual  P-P  and  Q-Q  plots  for  BTRA-BC  Time  to  Solution  and 
BTRA-BC  quality  appeared  roughly  normal.  There  was  enough  deviation  in  all  the  plots 
that  a  Kolmogorov-Smirnov  (K-S)  test  was  conducted  of  the  hypothesis  that  the  residuals 
were  drawn  from  a  nonnal  distribution.  The  K-S  test  did  not  reject  the  hypothesis  that 
the  residuals  were  drawn  from  a  normal  distribution.  Note  that  the  K-S  test  results  must 
be  viewed  with  caution,  because  the  underlying  assumption  of  independent  observations 
is  not  met  in  this  case. 


TASK 

SIGNIFICANCE 

Identifying  Potential 

0.092 

Avenues  of  Approach 

Identifying  Choke  Points 

0.100 

Determining  Hostile  Force 

0.015 

Engagement  Areas 

Determining  Battle 

0.025 

Positions 

Determining  Assault/Hide 

0.057 

Positions 

Determining  Ambush  Sites 

0.111 

Ability  to  Answer 

0.026 

Commander’s  Questions 

Overall 

0.049 

Table  3:  Perception  by  Task 


We  conducted  a  simulation  to  determine  what  the  potential  significance  of  the 
analyses  would  be  if  we  had  an  additional  eight  participants  with  the  same  characteristics 


i.e.  with  responses  in  the  same  statistical 
distributions.  Our  simulation  generated 
ten  sets  of  eight  additional  data  points  for 
both  the  BTRA-BC  and  CSE  conditions 
for  Time  to  Completion,  Quality,  and 
Terrain  Understanding.  Each  set  of 
points  was  generated  randomly  from  the 
nonnal  distribution  specific  to  each 


ANOVA 

Time  to 

Terrain 

Trial 

Completion 

Quality 

Understanding 

1 

0.628 

0.014 

0.177 

2 

0.960 

0.015 

0.325 

3 

0.788 

0.000 

0.122 

4 

0.480 

0.000 

0.669 

5 

0.291 

0.108 

0.366 

6 

0.095 

0.025 

0.213 

7 

0.602 

0.337 

0.669 

8 

0.391 

0.021 

0.934 

9 

0.344 

0.025 

0.189 

10 

0.546 

0.013 

0.132 

Table  4:  Simulation  Results 
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condition  and  type  of  data.  With  10  data  sets  from  a  total  of  sixteen  participants  (eight 
real  and  eight  simulated),  we  repeated  the  ANOVA  analyses  described  in  the  results 
section  above.  Table  4  contains  the  results  of  the  simulation  runs.  For  Quality,  eight  of 
ten  p-values  are  less  than  the  traditional  0.05  indicating  that  if  we  continued  the 
experiment  with  eight  additional  subjects  we  would  likely  achieve  statistically  significant 
results  confirming  that  participants  using  BTRA-BC  produced  higher  quality  products 
(hypothesis  2).  Our  simulation  results  suggest  that  adding  eight  additional  subjects  is 
unlikely  to  yield  strong  statistical  evidence  regarding  Hypothesis  1  (time  to  completion) 
or  Hypothesis  3  (terrain  understanding). 

8.  Conclusions 

The  first  run  of  the  experiment  was  conducted  in  April  of  2008.  Although  only 
eight  of  the  estimated  sixteen  required  participants  were  available,  the  results  are 
encouraging.  The  statistically  significant  results  from  our  preliminary  analysis  are: 

1.  Participants  using  CSE  w/  BTRA-BC  produce  better  quality  plans. 
Statistical  evidence  supports  this  and  there  is  also  strong  evidence  that  the 
produced  better  plans  with  respect  to  the  areas  supported  by  the  TSOs. 

2.  Participants  demonstrated  no  loss  of  Terrain  Understanding  due  to  system 
automation.  The  participants’  Terrain  understanding  using  CSE  w/ 
BTRA-BC  was  equal  to  or  better  when  than  their  terrain  understanding 
when  using  CSE  alone. 

3.  There  is  evidence  that  there  were  two  learning  effects.  On  average, 
participants  finished  the  tasks  in  less  time  on  the  second  trial.  This 
indicates  that  they  continued  to  leam  about  analyzing  the  problem 
throughout  the  trials,  but  this  effect  is  not  related  to  the  systems  used. 
There  was  also  an  inverse  learning  effect  in  that  participants  who  used 
CSE  w/  BTRA-BC  first  took  longer  when  using  CSE  alone. 

4.  The  participants  believed  that  using  CSE  w/  BTRA-BC  improved  the 
quality  of  their  plans,  made  their  planning  easier  and  that  overall  CSE  w/ 
BTRA-BC  was  superior  to  CSE  alone. 
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Background 


Map  is  focal  point  of  the  command  post 

Automated  geospatial  support  tools  are 
rapidly  penetrating  all  command  levels 

Empirical  research 
is  needed  to: 

-  Evaluate  military  value 
of  emerging  tools 

-  Prioritize  future  tool 
development 
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Why  Conduct  Experiments? 


•  Most  military  R  &  D  tests  to  requirements 

•  Hypothesis  driven  to  test  value-added 

•  Statistically  significant  results 

-  Quantitative  not  just  qualitative  feedback 

•  Answer  questions: 

-  What  is  the  value  added  for  the  warfighter? 

-  Does  the  product  meet  operational  needs? 

-  How  can  the  product  be  improved? 


Purpose  of  Research  Program 

Sponsored  by 

-  U.S.  Army  Engineer  Research  and  Development  Center 
(ERDC) 

-  U.S.  Army  GeospatialCenter  (AGC) 

Purpose: 

-  Assess  the  value-added  to  Military  Decision  Making  from 
use  of  Geospatial  Decision  Support  Products  (GDSPs) 

-  Evaluate  contribution  of  the  Battlespace  Terrain  Reasoning 
and  Awareness  -  Battle  Command  (BTRA-BC)  suite  of 
geospatial  reasoning  tools 


BTRA-BC  II 


Objective: 

-  Empower  commanders,  soldiers,  and  systems  with  information  that 
allows  them  to  understand  and  incorporate  the  impacts  of  terrain  and 
weather  on  their  functional  responsibilities  and  processes 


•  Products 

Information  and  knowledge  products  that  capture  integrated  terrain 
and  weather  effects 

Tactical  Spatial  Objects  (TSOs)  -  Predictive  decision  tools  that  exploit 
these  products 


•  Some  BTRA-BC  products  have  been  fielded  in  the  U.S. 
Army’s  Digital  Topographic  Support  System  (DTSS) 

Used  by  U.S.  Army  for  terrain  analysis 


Current  Study 


•  Study  Obj  ective 

-  Assess  the  benefit  of  BTRA-BC  tools  to  military  planners 
in  a  complex  and  realistic  scenario 

-  Expand  on  results  of  previous  experiment  (presented  at  last 
year’s  ICCRTS) 

•  CO  A  generation  vs.  AA  recommendation 

•  Planners  vs.  terrain  analysts 

•  More  complex  scenario  and  tasks 

•  More  complex  decision-making 
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Mission: 

-  Move  to  seize  an  objective  in  the  presence  of  the  enemy 


•  Analyze  actual  terrain  data 

•  Plan  a  Course  of  Action  (COA) 

•  Mechanized  Battalion 


Primary  Hypotheses 


1 .  Participants  who  use  BTRA-BC  TSOs  will  produce  military  planning 
output  more  quickly 

2.  Participants  who  use  BTRA-BC  TSOs  will  produce  a  higher  quality 
plans 

3.  Participants  who  use  BTRA-BC  TSOs  will  display  as  good  an 
understanding  of  the  impact  of  the  given  terrain  on  military  planning 

4.  The  quality  of  the  output  generated  with  BTRA-BC  TSOs  will  be  more 
uniform 

5.  There  will  be  little  or  no  learning  effect  due  to  evaluation  design 

6.  Participants  will  consider  using  BTRA-BC  TSOs  superior  when 
producing  a  plan  with  respect  to  speed,  quality,  ease  and  overall 


UNIVERSITY 


C4I  Center 


Study  Design 


•  Environment 

-  Commander’s  Support  Environment  (CSE) 

•  Developmental  C2  system 

•  Originally  a  DARPA  initiative 
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Three  independent  variables 


-  System  used  (with  and  without  BTRA-BC  TSOs) 

-  System  Order  (which  system  was  used  first) 

-  Scenario  Order  (Which  of  two  near  identical  scenarios 
was  used  first) 


Study  Design 


•  Within  Participants  design  with  respect  to  System  used: 

Each  subject  will  solve  a  planning  scenario  in  both  conditions  (with 
and  without  BTRA  TSOs) 

•  Between  Participants  design  with  respect  to 

-  System  Order 

-  Scenario  Order 

Design  was  counterbalanced  on  scenario  order  and  system  order 

•  Study  design  will  maintain  the  required  statistical  power  and 
minimize  the  number  of  participants 


^GEORGE 


C4I  Center 
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Training  prior  to  trials 

-  CSE  (4  hours)  and 

-  BTRA-BC  (2  hours) 


UNIVERSITY 


Study  Design  (cont) 


-Participants 

U.S.  Army  Majors,  Lt.  Colonels,  Colonels 

•  Planning  experience 

•  Comfortable  with  digital  systems 
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-  Experience 

•  Questionnaire 

•  Ranked  and  grouped  by  experience 

•  Randomly  assigned  to  groups 


-  Anonymous 

•  Randomly  assigned  participant  numbers 

•  Randomly  assigned  data  designators 


Experimental  Tasks 

•  The  evaluation  scenario  began  with  analysis  of  specific  terrain  and 
continued  to  the  point  of  generating  a  plan  of  movement  and  a  Course  of 
Action  (CO A). 

•  Specific  tasks  : 

-  Digital  Plan 

•  Plan  movement 

-  Identify  Mobility  Corridors  (MC) 

-  Categorize  Mobility  Corridors  by  size 

-  Group  Mobility  Corridors  to  form  potential  Avenues  of  Approach 

-  Identify  Choke  Points  on  Avenues  of  Approach 

-  Calculate  travel  times  and  coordinate  simultaneous  arrival 

•  Identify  Engagement  Areas 

•  Identify  Battle  Positions 

•  Identify  Ambush  Sites 

•  Identify  Named  Areas  of  Interest  (NAI) 

•  Generate  battalion  graphics  including  subordinate  echelon  Areas  of  Responsibility 


BTRA-BC  Tier  1  TSOs 
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Mobility 

Corridors 

Route 

Chokepoints 


BTRA-BC  Tier  2  TSOs 
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Battle 

Positions 


Hide 

Positions 


Engagement 

Area 


Experimental  Tasks  (cont) 


15 


Specific  tasks  (cont) 

-  Operation  Order 
•  Commander’s  Intent 


•  Concept  of  Operations 

-  Explanation  of  graphics 

-  Impact  of  terrain  on  mission 

-  Terrain  Understanding  Questionnaire 

-  System  Comparison  Questionnaire 


UNIVERSITY 


Measures 


•  Time  to  complete  scenario  (HI,  H5) 

-  Objective 

-  Significant  in  prior  experiment 

-  Possibly  less  significant  in  more  complex  planning 

•  Quality  of  solutions  as  judged  by  expert  evaluators  (H2,  H4,  H5) 

-  Subjective 

-  45  criteria  in  15  categories 

-  Independent  SMEs 
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Scores  on  a  questionnaire  evaluating  subject  understanding  of  the  terrain 
(H3,  H5) 

Scores  on  a  questionnaire  evaluating  subjective  perception  of  w/  BTRA- 
BC  (H6) 

-  Scale  Normal  and  Reversed 


Preliminary  Results:  Plan  Quality  (H2) 


There  is  statistical  evidence  that: 

Participants  produced  a  higher  quality  output  using 
CSE  w/  BTRA-BC  [F(l,4)  =  5.35,  p  =  0.08] 


Performed  a  repeated-measures 
ANOVA  on  the  average  of  all  13 
measures  of  plan  quality 

Approached  traditional  0.05 
significance  level 

No  other  effects  appeared 
significant. 


w/  BTRA-BC 


T 


w/o  BTRA-BC 


-w /  BTRA-BC  first 


-w/o  BTRA-BC  first 


System  use  order 

Error  Bars:  95%  Cl 


GEORGE 

_  _ASON  ,  __ 

university  C4I  Center 


Preliminary  Results:  TSO-related  Measures  (H2) 


•  Participants  produced  a  higher  quality  output  using 
measures  directly  related  to  BTRA-BC  TSOs  [F(l,4)  = 

12.62,  p  =  0.02] 


Performed  a  repeated-measures 
(ANOVA)  on  the  average  of  the 
TSO  related  measures 

Possible  learning  effect  for  CSE 
w/o  BTRA-BC  [p  =  0.08](H5) 

No  other  significant  effects 


Preliminary  Results:  Terrain  Understanding  (H3,  H4) 


•  There  is  no  statistical  evidence  that  participants  knowledge 
of  the  impact  of  the  given  terrain  on  military  planning 
differed  when  using  CSE  w/  BTRA-BC  (H3) 


19 


Participants  who  used  CSE  w/ 

BTRA-BC  first  had  significantly 
less  variance  (more  uniformity) 
in  measures  of  their  terrain 
understanding  than  those  who 
used  CSE  w/o  BTRA  first  [F(l,7) 

=  0.10,  p  =  0.00]  (H4) 

CSE  w /  BTRA-BC  first  Var  =  0.25 

CSE  w/o  BTRA-BC  first  Var  =  2.46  -w/  BTRA-BC  first  -w/o  BTRA-BC  first 


w/o  BTRA-BC 


System  use  order 


Preliminary  Results:  Subjective  Perception  (H6) 


There  is  strong  statistical  evidence  that: 

1 .  Participants  believe  they  can  produce  an  output  of  higher  quality 
w/  BTRA-BC  than  w/o  BTRA-BC 


2.  Participants  believe  that  overall  CSE  with  BTRA-BC  was 
superior  to  CSE  w/o  BTRA-BC 


•  The  results  provide 
marginally  significant 
evidence  producing  a  plan 
using  CSE  with  BTRA-BC 
TSOs  was  easier  than  with 
BTRA-BC  TSOs. 

•  No  effect  for  speed 


Participants'  Subjective  Evaluation  of  BTRA-BC 
[95%  Confidence  Intervals] 


■  Quality  ■  Overall  ■  Ease  STD 
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Summary  (1  of  2) 

Preliminary  results  are  encouraging 
BTRA-BC  TSOs  improved  the  planning  process 

-  Participants  produced  a  higher  quality  output  using  CSE 
w/  BTRA-BC  when  all  measures  are  considered 

-  Participants  produced  a  higher  quality  output  using 
measures  directly  related  to  BTRA-BC  TSOs 


Participants  who  used  CSE  w/  BTRA  first  had 
significantly  less  variance  (more  uniformity)  in 
measures  of  their  terrain  understanding 


There  is  a  learning  effect  in  that  participants  who  used 
CSE  w/BTRA-BC  first  produced  higher  quality  output 
when  they  used  CSE  w/o  BTRA-BC 


"^GEORGE 


C4I  Center 
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Summary  (2  of  2) 


■ 


Participants  believe  they  can  produce  an  output  of 
higher  quality,  more  easily  and  that  overall  CSE  with 
BTRA-BC  was  superior  to  CSE  w/o  BTRA-BC 

-  Participants  did  not  generate  the  output  more  quickly 

-  Participants  knowledge  of  the  impact  of  the  given 
terrain  on  military  planning  was  not  degraded 

These  results  will  be  strengthened  when  data  from 
the  full  set  of  1 6  participants  is  analyzed. 
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Next  Experiment  in  the  Series 


•  Object:  Assess  the  value  of  Buckeye’s  4-inch 
resolution  imagery  and  DTED  5  elevation  data 

•  Examining  accuracy  of  data  vice  effectiveness  of 
tools 

•  Experimental  Design 

-  Platoon  /  reinforced  squad 

-  Iraqi  city  where  CIB 1  and  Buckeye  data  are  available 

-  Planning  task:  Evaluation  of  potential  sites  for  Vehicle 
Control  Point  (VCP) 

-  Environment  CSE 

-  Participants:  16  infantry  E6-E7  or  02-03  with  experience 
in-country 


GEORGE 


Questions? 


